### I use HSM backed SSH certs and so can you. [why?: keys can be stolen, certs expire!]
 
1. Get a YubiHSM2 @ https://www.yubico.com/products/hardware-security-module/
2. Follow this: https://github.com/YubicoLabs/yubihsm-ssh-tool [ Yes, you're going to have to install all the other yubico stuff too, yubico-connector, etc, ..] on your issuing machine, or airgapped machine.
3. Be content that you can now sign certificates with the HSM on the issuer/airgapped machine.
3. Update /etc/ssh/sshd_config on remote server to add:
    TrustedUserCAKeys /etc/ssh/ca.pub
    AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
4. Add principals here:
    ex:
    root@remoteserver:~/.ssh# cat /etc/ssh/auth_principals/cmc
    access-cmc
 
5. You'll want your user to be the key ID and the principal that you burn into the cert to be on the server as above, if that user is allowed to access it.
 
6. SSH to the box, see something like this in /var/log/auth.log if successful:
    Accepted publickey for cmc from X.X.X.X port 50625 ssh2: RSA-CERT ID cmc (serial 0) CA RSA
    SHA256:NL5ksk+u5P96nTD90X/c0v78B0IIhITq
 
 
Need to debug server side? - /usr/sbin/sshd -p 2222 -D -d -e
Use SSH -p 2222 to connect on this second ssh server. Review logs.
 
#### Add/Removing identities, working.
#### check i've got the cert loaded. cool beans.
 
cmc@cmc-pro example_keypair % ssh-add -l
2048 SHA256:82fd6WodUUJbBmIfNJLgZVV/G96DIhkbscXAY+38gmY cmc@cmc-pro (RSA)
2048 SHA256:82fd6WodUUJbBmIfNJLgZVV/G96DIhkbscXAY+38gmY cmc@cmc-pro (RSA-CERT)
 
##### Yes, it works.
cmc@cmc-pro example_keypair % ssh -i id_rsa-cert.pub cmc@remoteserver.net
Duo two-factor login for cmc
 
Enter a passcode or select one of the following options:
 
 1. Duo Push to XXX-XXX-6084
 2. Phone call to XXX-XXX-6084
 3. SMS passcodes to XXX-XXX-6084 (next code starts with: 2)
 
Passcode or option (1-3): ^CConnection to remoteserver.net closed.
 
### now i'll remove it
cmc@cmc-pro example_keypair % ssh-add -D
All identities removed.
 
### I'll try and add it.
cmc@cmc-pro example_keypair % ssh -i id_rsa-cert.pub cmc@remoteserver.net
Load key "id_rsa-cert.pub": invalid format
cmc@remotebox.net's password:
#### ^^^ Ah yes, very good. </sarcasm>
 
#### YOU HAVE TO ADD THE ORIGINAL RSA KEY OTHERWISE IT WONT IMPORT THE CERT AND DOES NOT DO IT DIRECTLY ON THE CERT FILE
cmc@cmc-pro example_keypair % ssh-add ./id_rsa
Identity added: ./id_rsa (cmc@cmc-pro)
Certificate added: ./id_rsa-cert.pub (cmc)
cmc@cmc-pro example_keypair % ssh-add -l
2048 SHA256:82fd6WodUUJbBmIfNJLgZVV/G96DIhkbscXAY+38gmY cmc@cmc-pro (RSA)
2048 SHA256:82fd6WodUUJbBmIfNJLgZVV/G96DIhkbscXAY+38gmY cmc@cmc-pro (RSA-CERT) <---- back
 
 
###### Success is yes
cmc@cmc-pro example_keypair % ssh -i id_rsa-cert.pub cmc@remoteserver.net
Duo two-factor login for cmc
 
Enter a passcode or select one of the following options:
 
 1. Duo Push to XXX-XXX-6084
 2. Phone call to XXX-XXX-6084
 3. SMS passcodes to XXX-XXX-6084 (next code starts with: 2)
 
Passcode or option (1-3): ^CConnection to remoteserver.net closed.
cmc@cmc-pro example_keypair %



### script the sign

cmc@cmc-pro yubihsm % cat sign_my_cert.sh
#!/bin/sh
# validity, -/+ 5hrs.
# ./sign_my_cert user

USER_ID="cmc"
USER_PRINCIPAL="access-cmc"
KEY_TO_SIGN=$1

# submit req + sign
sudo yubihsm-ssh-tool req -s ca_pub.pem -t timestamp.pem -I $USER_ID -n $USER_PRINCIPAL -V -5h:+5h $KEY_TO_SIGN.pub
yubihsm-shell -a sign-ssh-certificate -p password -i 10 --template-id 20 -A rsa-pkcs1-sha256 --in req.dat --out $1-cert.pub
echo ### Signed $1 ####
ssh-keygen -Lf $1-cert.pub
chmod 400 $1*
ssh-add $1
ssh-add -l
cmc@cmc-pro yubihsm %

### output

## gen demo cert

ssh-keygen -t rsa -N "" -f ./lol

## sign it

cmc@cmc-pro yubihsm % sh -x ./sign_my_cert.sh lol
+ USER_ID=cmc
+ USER_PRINCIPAL=access-cmc
+ KEY_TO_SIGN=lol
+ sudo yubihsm-ssh-tool req -s ca_pub.pem -t timestamp.pem -I cmc -n access-cmc -V -5h:+5h lol.pub
('Hash is:', '53d485f73e0a1ac47c1e3bcbeda249d040c7f1d0f1f9bf5b124a12dd8c0a81c8')
+ yubihsm-shell -a sign-ssh-certificate -p password -i 10 --template-id 20 -A rsa-pkcs1-sha256 --in req.dat --out lol-cert.pub
Using default connector URL: http://127.0.0.1:12345
Session keepalive set up to run every 15 seconds
Created session 0
+ echo

+ ssh-keygen -Lf lol-cert.pub
lol-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:HVVNHo2xFErv83FcbjxKESHHofBBYGhZJngjMuk+Qp8
        Signing CA: RSA SHA256:NL5ksk+u5P96nTD90X/c0v78B0IIhITqEEGWqairac0
        Key ID: "cmc"
        Serial: 0
        Valid: from 2020-04-08T20:34:55 to 2020-04-09T06:34:55
        Principals:
                access-cmc
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc
+ chmod 400 lol lol-cert.pub lol.pub 
+ ssh-add lol
Identity added: lol (cmc@cmc-pro)
Certificate added: lol-cert.pub (cmc)
+ ssh-add -l
2048 SHA256:HVVNHo2xFErv83FcbjxKESHHofBBYGhZJngjMuk+Qp8 cmc@cmc-pro (RSA)
2048 SHA256:HVVNHo2xFErv83FcbjxKESHHofBBYGhZJngjMuk+Qp8 cmc@cmc-pro (RSA-CERT)
cmc@cmc-pro yubihsm %